Haptic effect generation for space-dependent content

ABSTRACT

Systems and methods for authoring and encoding haptic effects are provided for space-dependent content, such as 360-degree videos, three-dimensional videos, or virtual or augmented reality contents. The systems and methods can generate one or more haptic layers for encoding or modifying haptic effects for the content.

RELATED APPLICATIONS

The present patent application claims priority to U.S. patentapplication Ser. No. 16/181,235 filed Nov. 5, 2018, which issued as U.S.Pat. No. 10,720,189 on Jul. 21, 2020. Patent application Ser. No.16/181,235 claims priority to U.S. patent application Ser. No.15/392,744 filed Dec. 28, 2016, which issued as U.S. Pat. No. 10,147,460on Dec. 4, 2018. The above patents are assigned to the assignee hereofand filed by the inventors hereof and are incorporated by referenceherein.

BACKGROUND

Haptics is a tactile and force feedback technology that takes advantageof a user's senses by haptic effects such as vibrations, motions, andother forces and stimulations. Devices, such as mobile devices, gamingdevices, touchscreen devices, and personal computers, can be configuredto generate haptic effects. Haptic feedback can provide kinestheticfeedback (such as active and resistive force feedback) and/or tactilefeedback (such as vibration, vibrotactile feedback, texture, heat, etc.)to a user. Haptic effects may be useful to alert the user to specificevents or to provide realistic feedback to create greater sensoryimmersion within a simulated or virtual environment.

Devices can be configured to coordinate the output of haptic effectswith the output of other content, such as videos, games or other media.In some types of video content, such as a 360-degree video,three-dimensional video, or virtual reality video, a video scene canchange based on a user or viewer's perspective or point of interest, andit may be desirable to provide different haptic effects to differentobjects to which the user's view is directed in the scene.

SUMMARY

One aspect is a method of authoring haptic effects. The method includesreceiving a haptic track and a video texture map and generating hapticdata modifiers associated with haptic pixels. The haptic track hashaptic data defining operation of a haptic actuator. The video texturemap is representative of at least one frame of a video. The haptic trackis associated with the at least one frame of the video. The videotexture map has a plurality of video pixels. Each haptic pixel isassociated with at least one of the video pixels. The haptic datamodifiers are configured to modify at least a portion of the haptic dataand modify operation of the haptic actuator.

Another aspect is an apparatus for authoring haptic effects. Theapparatus includes a processing device configured to control operationof the apparatus, and one or more computer readable data storage mediastoring software instructions that, when executed by the processingdevice, cause the apparatus to: receive a haptic track and a videotexture map and generate haptic data modifiers associated with hapticpixels. The haptic track has haptic data defining operation of a hapticactuator. The video texture map is representative of at least one frameof a video. The haptic track is associated with the frame of the video,and the video texture map has a plurality of video pixels. Each hapticpixel is associated with at least one of the video pixels. The hapticdata modifiers are configured to modify at least a portion of the hapticdata and modify operation of the haptic actuator.

Yet another aspect is a method of generating haptic effects. The methodincludes receiving a haptic track including haptic data, the haptic datadefining operation of a haptic actuator; receiving a plurality of hapticpixels including haptic data modifiers, each haptic pixel associatedwith at least one video pixel, the at least one video pixel included ina video texture map representative of at least one frame of a video;identifying a point of interest in the frame of the video, the point ofinterest including a target video pixel in the frame of the video;determining a target haptic pixel among the haptic pixels, the targethaptic pixel corresponding to the target video pixel in the frame of thevideo; obtaining a target haptic data modifier included in the targethaptic pixel; modifying the haptic data in the haptic track based on thetarget haptic data modifier; and controlling operation of a hapticactuator based on the modified haptic data.

Yet another aspect is an apparatus for generating haptic effects. Theapparatus includes one or more actuators, an actuator drive circuitconfigured to operate the actuator, a sensor configured to detect apoint of interest, and a processing device coupled to the actuator drivecircuit and the sensor. The processor is configured to: receive a haptictrack including haptic data, the haptic data defining operation of ahaptic actuator; receive a plurality of haptic pixels including hapticdata modifiers, each haptic pixel associated with at least one videopixels, the at least one video pixels included in a video texture maprepresentative of a frame of a video; identify, using the sensor, apoint of interest in the frame of the video, the point of interestincluding a target video pixel in the frame of the video; determine atarget haptic pixel among the haptic pixels, the target haptic pixelcorresponding to the target video pixel in the frame of the video;obtain a target haptic data modifier included in the target hapticpixel; modify the haptic data in the haptic track based on the targethaptic data modifier; generate a control signal based on the modifiedhaptic data; and transmit the control signal to the actuator drivecircuit, the control signal enabling the actuator drive circuit tocontrol the actuator.

Yet another aspect is a method of generating haptic effects. The methodincludes receiving a plurality of haptic pixels, each haptic pixelassociated with at least one video pixel and including haptic data, thehaptic data defining operation of a haptic actuator, the at least onevideo pixel included in a video texture map representative of at leastone frame of a video; identifying a point of interest in the at leastone frame of the video, the point of interest including a target videopixel in the at least one frame of the video; determining a targethaptic pixel among the haptic pixels, the target haptic pixelcorresponding to the target video pixel in the at least one frame of thevideo; obtaining a target haptic data included in the target hapticpixel; and controlling operation of a haptic actuator based on thetarget haptic data.

Yet another aspect is an apparatus for generating haptic effects. Theapparatus includes one or more actuators, an actuator drive circuitconfigured to operate the actuator, a sensor configured to detect apoint of interest, and a processing device coupled to the actuator drivecircuit and the sensor. The processor is configured to: receive aplurality of haptic pixels, each haptic pixel associated with at leastone video pixel and including haptic data, the haptic data definingoperation of a haptic actuator, the at least one video pixel included ina video texture map representative of a frame of a video; identify apoint of interest in the frame of the video, the point of interestincluding a target video pixel in the frame of the video; determine atarget haptic pixel among the haptic pixels, the target haptic pixelcorresponding to the target video pixel in the frame of the video;obtain a target haptic data included in the target haptic pixel; andgenerate a control signal based on the target haptic data; and transmitthe control signal to the actuator drive circuit, the control signalenabling the actuator drive circuit to control the actuator.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for authoring and rendering hapticeffects in accordance with an exemplary embodiment of the presentdisclosure.

FIG. 2 is a flowchart illustrating an exemplary method for operating thesystem of FIG. 1.

FIG. 3 is a block diagram of a possible embodiment of a haptic effectauthoring device as illustrated in FIG. 1.

FIG. 4 is a block diagram illustrating an example operation of thehaptic effect authoring device.

FIG. 5 is a flowchart illustrating a possible exemplary method ofoperating the haptic effect authoring device.

FIG. 6 illustrates an exemplary method of playing a video.

FIG. 7 illustrates an exemplary method of associating a video texturemap with a haptic texture map.

FIG. 8 is one possible example data structure of haptic pixels.

FIG. 9 is a flowchart illustrating an exemplary method for generatinghaptic data modifiers.

FIG. 10 illustrates one possible embodiment of a user interface forreceiving a user input of haptic data modifier values.

FIG. 11 illustrates a block diagram of one of many possible embodimentsof a haptic enabled apparatus as illustrated in FIG. 1.

FIG. 12 is a flowchart illustrating an exemplary method for generatinghaptic effects.

FIG. 13 illustrates an exemplary method of determining a point ofinterest in a video frame.

FIG. 14 is a flowchart of an example method for identifying the point ofinterest as illustrated in FIG. 13.

FIG. 15 illustrates an exemplary process of rendering a video contentwith haptic effects.

FIG. 16 illustrates that a target haptic pixel is determined in a set ofhaptic pixels.

FIG. 17 illustrates a data structure of another exemplary set of hapticpixels.

FIG. 18 is a possible exemplary method for generating haptic effectsusing the haptic pixels of FIG. 17.

DETAILED DESCRIPTION

Various embodiments will be described in detail with reference to thedrawings, wherein like reference numerals represent like parts andassemblies throughout the several views. Reference to variousembodiments does not limit the scope of the claims attached hereto.Additionally, any examples set forth in this specification are notintended to be limiting and merely set forth some of the many possibleembodiments for the appended claims.

Whenever appropriate, terms used in the singular also will include theplural and vice versa. The use of “a” herein means “one or more” unlessstated otherwise or where the use of “one or more” is clearlyinappropriate. The use of “or” means “and/or” unless stated otherwise.Terms such as “comprise,” “comprises,” “comprising,” “include,”“includes,” “including,” “such as,” “has,” and “having” areinterchangeable and not intended to be limiting. For example, the term“including” shall mean “including, but not limited to.”

In general, the present disclosure relates to systems and methods forauthoring and encoding haptic effects for content, such as a video. Thesystems and methods of the present disclosure allow the authoring andencoding of space-dependent (or orientation- or direction-dependent)content, such as 360-degree videos, three-dimensional videos, or virtualor augmented reality contents. The present disclosure allows localizinghaptic effects to one or more particular locations or objects in suchcontent.

A haptic effect or haptic feedback can be any type of tactile sensationdelivered to a person. In some embodiments, the haptic effect embodiesinformation such as a cue, notification, feedback or confirmation of auser's interaction with a haptic-enabled article, or a more complexmessage or other information. In alternative embodiments, the hapticeffect can be used to enhance a user's interaction with a device bysimulating a physical property or effect such as friction, flow, anddetents.

The systems and methods of the present disclosure can generate one ormore haptic layers, such as haptic texture maps. The haptic layer has aplurality of haptic pixels which contain information to encode or edithaptic effects. The haptic pixels include information for defining oradjusting haptic effects associated with content, such as video content.In one possible embodiment, each haptic pixel can include a coordinatefor associating the haptic pixel with one or more corresponding videoframes or video texture maps.

Each haptic layer can be created based on at least one of the videoframes that are streamed to generate a video. In certain examples, thehaptic layers can be created and used at the same frame rates as thevideo frames. In other examples, the haptic layers can be used at adifferent frame rate that the video frames, so that a single hapticlayer encodes haptic effects for a plurality of video frames.

In certain examples, each haptic layer has haptic pixels each of whichis associated with at least one pixel in a corresponding video frame orvideo texture map corresponding one or more video frames. Each hapticpixel in a particular haptic layer can encode or edit haptic effectsassociated with one or more pixels in at least one video frame or videtexture map corresponding to that haptic layer. As such, because thehaptic pixels in a haptic layer may have different information,parameters, and/or values, the haptic effects can be localized asdesired in one or more video frame corresponding to the haptic layer.

Where one or more haptic tracks are used to provide haptic data forgenerating haptic effects, the haptic layer includes information formodifying the haptic data to adjust the haptic effects. For example,where a haptic track includes haptic data for generating an amplitudesignal, each haptic pixel in a haptic layer has a haptic data modifierfor changing the haptic data, thereby adjusting the amplitude signal. Byproviding different haptic data modifiers to the haptic pixels in thehaptic layer, different haptic effects (e.g., different amplitudes ofhaptic effect) can be provided to different locations in one or morevideo frames associated with the haptic layer. The system of presentdisclosure may provide a user interface for enabling a user to customizethe haptic data modifiers.

In certain embodiments, such haptic tracks can be encoded in a hapticlayer such that the haptic pixels in the haptic layer include hapticdata used to generate haptic effects. In this configuration, duringplayback of a video content, haptic effects can be generated usinghaptic layers without requiring separate haptic tracks.

During playback of a video content, a point of interest in the videocontent can be determined at a particular time and used to index anappropriate haptic effect associated with the point of interest. Incertain embodiment, the point of interest is the orientation vector ofthe viewer. The point of interest can be represented as, or associatedwith, a haptic pixel in a haptic layer at that time. The haptic effectassociated with the point of interest is rendered in synchronizationwith the video playback (i.e., the video frame at the time).

Although many examples described in the present disclosure relate to a360-degree video, it should be understood that the teachings of thepresent disclosure may also include other applications in which hapticeffects can be applied to playback of other contents, such as a virtualor augmented reality experience in which a user can look around.

FIG. 1 is a block diagram of a system for authoring and rendering hapticeffects in accordance with an exemplary embodiment of the presentdisclosure. The system is generally designated with reference number 100and includes a haptic effect authoring device 102 and a haptic enabledapparatus 104. In at least some embodiments, the haptic enabledapparatus 104 includes a haptic effect rendering device 106 and acontent presentation device 108.

The haptic effect authoring device 102 operates to modify haptic datafor generating haptic effects. In certain embodiments, the haptic datacan be included in one or more haptic tracks 112 and define operation ofone or more haptic actuators. In at least some embodiments, the hapticeffect authoring device 102 receives an input of one or more parametersthat define or edit characteristics of haptic effect, such as amplitude,frequency, waveform, sharpness, and other possible attributes of hapticeffect, and modifies the haptic data based on the parameters. An exampleof the haptic effect authoring device 102 is described in more detailherein, including the description with reference to FIGS. 3-10.

In at least some embodiments, the haptic effect authoring device 102generates a set of haptic pixels 110. In certain embodiments, the hapticeffect authoring device 102 can receive and use one or more haptic track112 and content data 114 to generate the haptic pixels 110.

The haptic pixels 110 include data defining or editing haptic effectsassociated with the content data 114. For example, where the contentdata 114 is represented as data pixels in one or more content datatexture maps (such as video texture maps as described herein), each ofthe haptic pixels 110 includes data defining a haptic effect associatedwith at least one of the data pixels in the content data texture maps.As described herein, the haptic pixels 110 can be represented as ahaptic texture map, as described in more detail herein, including thedescription associated with FIG. 7.

The haptic enabled apparatus 104 operates to generate haptic effectswhile playing back the content based on the content data 114. In certainembodiments, the haptic enabled apparatus 104 receives the haptic pixels110 and the haptic track 112 to render the haptic effects. An example ofthe haptic enabled apparatus 104 is further described in more detailherein, including the description with reference to FIG. 11.

The haptic effect rendering device 106 in the haptic enabled apparatus104 is configured to render haptic effects associated with the contentpresented using the content presentation device 108. In certainexamples, the haptic effect rendering device 106 generates hapticeffects while the content presentation device 108 presents the contentbased on the content data 114. Where the content data 114 is a videocontent, the content presentation device 108 displays the video contentbased on the content data 114, and the haptic effect rendering device106 renders the haptic effects corresponding to the video content.

The content presentation device 108 operates to present content based onthe content data 114. The content presentation device 108 can be ofvarious types, depending on the type of content data 114. For example,where the content data 114 includes video content data, the contentpresentation device 108 includes a display device (as illustrated inFIG. 11) for displaying video content. In other examples, where thecontent data 114 includes audio content data, the content presentationdevice 108 includes an audio playback device.

As further described herein including the description with reference toFIGS. 12 and 13, the haptic effect rendering device 106 further includesone or more sensors, such as an orientation sensor, for identifying apoint of interest in the content data during rendering. The point ofinterest can be one or more of various points in the video frame. Insome embodiments, the point of interest is a point at which a user orviewer looks in the video frame. In other embodiments, the point ofinterest is a point at which a user or viewer positions a pointer orcursor on a display device presenting the video frame. Such a point canbe referred to herein as a user perspective vector or look vector.

In some possible embodiments, the haptic effect rendering device 106 andthe content presentation device 108 are integrated in a single device.In such a configuration, the point of interest, such as the look vector,can be identical at the haptic effect rendering device 106 and thecontent presentation device 108. In other possible embodiments, thehaptic effect rendering device 106 can be separate from the contentpresentation device 108. For example, the haptic effect rendering device106 can be implemented as a hand-held controller while the contentpresentation device 108 is configured as a headset. In thisconfiguration, the point of interest, which, for example, can beselected using the hand-held controller, may be different from the lookvector, which represents a user's perspective using, for example, theheadset.

Although the haptic enabled apparatus 104 is described in theillustrated embodiment as including the haptic effect rendering device106 and the content presentation device 108, it should be understoodthat either or both of the haptic effect rendering device 106 and thecontent presentation device 108 can be configured separately from thehaptic enabled apparatus 104.

Referring still to FIG. 1, the haptic track 112 includes haptic dataassociated with haptic effects. The haptic data defines parameters forgenerating haptic drive signals for driving one or more hapticactuators. In exemplary embodiments, such parameters relate to, or areassociated with, characteristics of the haptic drive signals such asamplitude, frequency, phase, inversion, duration, waveform, attack time,rise time, lag or lead time relative to an event, and fade time. Thehaptic drive signals are applied to one or more haptic actuators tocause one or more haptic effects.

The content data 114 is data used to generate content which can bedelivered to a user via the content presentation device 108. In thisdocument, the content data 114 is primarily described as data for avideo content, such as a 360-degree video. In other embodiments,however, the teaching of the present disclosure can similarly be appliedto other types of content, such as three-dimensional videos, virtual oraugmented reality contents, and audio contents.

In the illustrated embodiments, the haptic effect authoring device 102generates the haptic pixels 110 based on the content data 114 and thehaptic track 112. In this configuration, the haptic enabled apparatus104 can receive both the haptic track 112 and the haptic pixels 110 togenerate haptic effects associated with the content data 114. In otherembodiments, the haptic effect authoring device 102 can generate thehaptic pixels 110 based on the content data 114, without requiring thehaptic track 112. In such embodiments, the haptic data, which would beotherwise included in the haptic track, are encoded in the haptic pixels110, and the haptic enabled apparatus 104 can render haptic effectsusing the haptic pixels 110.

FIG. 2 is a flowchart illustrating an exemplary method 130 for operatingthe system 100 of FIG. 1. In certain embodiments, the system 100generally operates to perform two steps to provide haptic effectsassociated with content presented to a user. First, the system 100 isconfigured and used to enable authoring haptic effects for content(operation 132), and render the haptic effects as the content ispresented (operation 134). In some embodiments, the operation 132 can beperformed using the haptic effect authoring device 102 and otherassociated devices in the system 100, and the operation 134 can beperformed using the haptic effect rendering device 106 and otherassociated devices in the system. An example of the operation 132 isillustrated and described with reference to FIGS. 3-10, and an exampleof the operation 134 is illustrated and described with reference toFIGS. 11-15.

FIG. 3 is a block diagram of a possible embodiment of the haptic effectauthoring device 102 as illustrated in FIG. 1. In this embodiment, thehaptic effect authoring device 102 includes a bus 140, a processor 142,an input/output (I/O) controller 144, memory 146, a network interfacecontroller (NIC) 148, and a user interface 150. The bus 140 includesconductors or transmission lines for providing a path to transfer databetween the components in the device 102 including the processor 142,the I/O controller 144, the memory 146, and the NIC 148. The bus 140typically comprises a control bus, address bus, and data bus. However,the bus 140 can be any bus or combination of busses, suitable totransfer data between components in the device 102.

The processor 142 can be any circuit configured to process informationand can include any suitable analog or digital circuit. The processor142 also can include a programmable circuit that executes instructions.Examples of programmable circuits include microprocessors,microcontrollers, application specific integrated circuits (ASIC),programmable gate arrays (PLA), field programmable gate arrays (FPGA),or any other processor or hardware suitable for executing instructions.In various embodiments, the processor 142 can be a single unit or acombination of two or more units. If the processor 142 includes two ormore units, the units can be physically located in a single controlleror in separate devices.

The I/O controller 144 is circuitry that monitors operation of thedevice 102 and peripheral or external devices such as the user interface150. The I/O controller 144 also manages data flow between the device102 and the peripheral devices and frees the processor 142 from detailsassociated with monitoring and controlling the peripheral devices.Examples of other peripheral or external devices with which the I/Ocontroller 144 can interface includes external storage devices;monitors; input devices such as keyboards and pointing devices; externalcomputing devices; antennas; other articles worn by a person; and anyother remote devices.

The memory 146 can include volatile memory such as random access memory(RAM), read only memory (ROM), electrically erasable programmable readonly memory (EEPROM), flash memory, magnetic memory, optical memory, orany other suitable memory technology. The memory 146 also can include acombination of volatile and nonvolatile memory.

The memory 146 can store a number of program modules for execution bythe processor 142, including a user input acquisition module 152, a dataacquisition module 154, a haptic authoring module 156, and acommunication module 158. Each module is a collection of data, routines,objects, calls, and other instructions that perform one or moreparticular task. Although certain modules are disclosed herein, thevarious instructions and tasks described herein can be performed by asingle module, different combinations of modules, modules other thanthose disclosed herein, or modules executed by remote devices that arein communication, either wirelessly or by wire, with the device 102.

The user input acquisition module 152 are instructions that, whenexecuted by the processor 142, cause the processor 142 to receive userinputs of one or more parameters associated with haptic effects orhaptic effect modifiers. The user input acquisition module 152 cancommunicate with the input device 162 of the user interface 150 andenable a user to input such parameters through the input device 162. Byway of example, the user input acquisition module 152 provides agraphical user interface on a display screen (i.e., the input device162) that allows a user to enter or select one or more parameters forhaptic effects. An example method of operating the user inputacquisition module 152 is illustrated in more detail with reference toFIGS. 9 and 10.

The data acquisition module 154 are instructions that, when executed bythe processor 142, cause the processor 142 to receive data used toauthor haptic effects. Such data can be used by the haptic authoringmodule 156. In the specific embodiment discussed herein, the datareceived include information about the haptic track 112 and the contentdata 114.

The haptic authoring module 156 are instructions that, when executed bythe processor 142, cause the processor 142 to generate the haptic pixels110. In at least some embodiments, the haptic authoring module 156communicates with the user input acquisition module 152 to receive userinputs and with the data acquisition module 154 to receive data usableto author haptic effects, such as the haptic track 112 and the contentdata 114.

The communication module 158 facilitates communication between thedevice 102 and remote devices. Examples of remote devices includecomputing devices, sensors, actuators, networking equipment such asrouters and hotspots, vehicles, exercise equipment, and smartappliances. Examples of computing devices include servers, desktopcomputers, laptop computers, tablets, smartphones, home automationcomputers and controllers, and any other device that is programmable.The communication can take any form suitable for data communicationincluding communication over wireless or wired signal or data paths. Invarious embodiments, the communication module may configure the device102 as a centralized controller of the system 100 or other remotedevices, as a peer that communicates with other computing devices orother remote devices, or as a hybrid centralized controller and peersuch that the controller can operate as a centralized controller in somecircumstances and as a peer in other circumstances.

Alternative embodiments of the program modules are possible. Forexample, some alternative embodiments might have more or fewer programmodules than the user input acquisition module 152, the data acquisitionmodule 154, the haptic authoring module 156, and the communicationmodule 158. In some possible embodiments, one or more of the programmodules are in remote devices such as remote computing devices or otherwearable articles.

Referring still to FIG. 3, the network interface controller (NIC) 148 isin electrical communication with a network 168 to provide communication(either wireless or wired) between the device 102 and remote devices.Communication can be according to any wireless transmission techniquesincluding standards such as Bluetooth, cellular standards (e.g., CDMA,GPRS, GSM, 2.5G, 3G, 3.5G, 4G), WiGig, IEEE 802.11a/b/g/n/ac, IEEE802.16 (e.g., WiMax). The NIC 148 also can provide wired communicationbetween the device 102 and remote devices through wired connectionsusing any suitable port and connector for transmitting data andaccording to any suitable standards such as RS 232, USB, FireWire,Ethernet, MIDI, eSATA, or thunderbolt.

The user interface 150 can include an input device 162 and an outputdevice 164. The input device 162 includes any device or mechanismthrough which a user can input parameters, commands, and otherinformation into the haptic effect authoring device 102. In addition,the input device 162 can also receive haptic tracks and content data,such as video tracks, for editing. Examples of input device 162 includetouchscreens, touch sensitive surfaces, cameras, mechanical inputs suchas buttons and switches, and other types of input components. The inputdevice 162 can also include removable memory readers for portablememory, such as flash memory, magnetic memory, optical memory, or anyother suitable memory technology. The output device 164 includes anydevice or mechanism that presents information to a user in variousformats, such as visual and audible formats. Examples of output device164 include display screens, speakers, lights, and other types of outputcomponents. The output device 164 can also include removable memoryreaders. In one embodiment, the input device 162 and the output device164 are integrally formed, such as a touch-sensitive display screen.

FIG. 4 is a block diagram illustrating an example operation of thehaptic effect authoring device 102. In at least some embodiments, thehaptic effect authoring device 102 receives a haptic track 112 andcontent data 114, and uses the haptic track 112 and the content data 114to generate a set of haptic pixels 110.

As described above, the haptic track 112 includes haptic data associatedwith haptic effects and defining operation of one or more hapticactuators. The content data 114 includes information for generatingcontent delivered using a content playback device, such as the contentpresentation device 108.

In the illustrated embodiment, video content is rendered with desiredhaptic effects. For example, the video content includes a sphericalvideo or 360-degree video. A 360-degree video is a video recording wherea view in every direction is recorded at the same time usingphotographic technology, such as a collection of cameras and anomnidirectional camera. The resulting footage then can be stitched toform a single rectangular video which can be projected onto a sphere.During playback, a view can control a viewing direction, which resemblesa panorama. In some possible embodiments, a 360-degree video ismonoscopic, which is viewed as a flat image on a singular surface. Inother possible embodiments, a 360-degree video is a stereoscopic video,which is viewed as two images directed individually to each eye.

The 360-degree video can be played back on various devices. For example,when a 360-degree video is viewed on personal computers, the mouse orother pointing devices (e.g., a trackballs, joystick, pointing stick,WillMote, finger tracking device, pen, or stylus) is used to pan aroundthe video by clicking and dragging. On mobile computing devices (e.g.,smartphone or tablet), internal sensors such as a gyroscope can be usedto pan the video based on the orientation of the device. In otherexamples, the video can be displayed to a user operating virtual realitydevices including virtual reality headsets, controllers, and otherdevices. The user's view point and/or movement can be used to pan aroundthe video.

In the illustrated embodiment where the content data 114 is for a videocontent, one or more video texture maps can be provided, which representone or more video frames. The video texture maps 212 can be used asvideo frames to render a video content on a display device. An exampleof the video texture map is further described herein, including thedescription with reference to FIGS. 5 and 6.

As described above, the haptic pixels 110 include information fordefining or adjusting haptic effects associated with the content data114. As described herein, in one possible embodiment, the haptic pixel110 can include a coordinate for associating the haptic pixel 110 withone or more corresponding video frames or video texture maps. An exampledata structure of a haptic pixel 110 is described herein, including thedescription with reference to FIG. 8. The haptic pixels 110 can berepresented as one or more haptic texture maps. In at least someembodiments, the haptic pixels 110 or the haptic texture maps thereofare not visually rendered. In other embodiments, the haptic texture mapscan be visually rendered, as illustrated herein including thedescription with reference to FIG. 9.

FIGS. 5-8 illustrate a possible exemplary method 200 of operating thehaptic effect authoring device 102. In the illustrated embodiment, themethod 200 includes operations 202, 204, and 206. In other embodiments,the method 200 can include additional operations with at least one ofthese operations.

At operation 202, the haptic effect authoring device 102 receives ahaptic track 112 and content data 114. At operation 204, the hapticeffect authoring device 102 generates haptic pixels 110 associated withvideo pixels 216 (FIG. 7) of a video texture map 212. At operation 206,the haptic effect authoring device 102 generates haptic effect modifiers232 (FIG. 8) associated with the haptic pixels 110. The haptic pixels110 correspond to the haptic pixels 110

In this embodiment, the content data 114 include a plurality of videotexture maps 212. Each video texture map can represent at least oneframe 218 of a video 214. The video 214 is played back on a video playerhaving a display screen, such as the content presentation device 108 ofthe haptic enabled apparatus 104, as illustrated in FIG. 1. The contentdata 114 includes information that the content presentation device 108can use to play the video.

As illustrated in FIG. 6, in some embodiments, the video 214 is playedback by displaying a plurality of frames 218. The plurality of frames218 can be displayed in series at a determined frame rate or temporalresolution. Some examples of temporal resolution range from 10 framesper second (FPS) to 500 FPS. Other temporal resolutions can also bepossible in other embodiments.

Referring still to FIG. 6, each of the video texture maps 212, which areincluded in the content data 114, can represent at least one frame 218of the video 214. In such an example as a 360-degree video, the videotexture maps 212 typically represent the frames 218, respectively, suchthat a single video texture map 212 is used as a single frame 218 of thevideo 214. In this configuration, the video texture maps 212 areidentical to the video frames 218 and thus can be also referred to asthe video frames 218. In other examples, at least one of the videotexture maps 212 is configured to represent two or more of the frames218 of the video 216. For example, where a video compression isperformed, each of the video texture maps 212 is associated with aplurality of frames 218 (such as two, three, four, etc.) such that, whenthe frames 218 are streamed at a first frame rate, the video texturemaps 212 are used at a second frame rate lower than the first framerate. By way of example, each video texture map 212 is configured torepresent three consecutive frames 218, so that when the frames 218 arestreamed at 90 FPS, the video texture maps 212 are used at 30 FPS. Inyet other examples, at least some of the video texture maps 212 areassociated with different numbers of frames 218. For example, a firstvideo texture map 212 corresponds to two frames 218 while a second videotexture map 212 represents only a single frame 218 different from thetwo frames 218.

The video texture maps 212 are two-dimensional images which can be usedas one or more frames 218 of the video 214. As described herein, thevideo texture maps 212 can also be identical to the video frames 218 insome examples. In at least some possible embodiments, various mappingtechniques can be used to project each video texture map 212 to at leastone frame 218 of the video 214. Example mapping processes include UVmapping, which projects a two-dimensional image to a three-dimensionalmodel's surface. The UV mapping process can use polygons that make up athree-dimensional object and have surface attributes, such as color,from a two-dimensional image, such as the video texture map 212. The UVmapping process involves assigning video pixels 216 (FIG. 7) in thevideo texture map 212 to surface mappings on the polygons. In otherexamples, other mapping methods can be used to associate the videotexture maps 212 to the frames 218 of the video 214.

Referring back to FIG. 5, at operation 204, the haptic effect authoringdevice 102 generates haptic pixels 110 associated with video pixels 216of a video texture map 212. In some embodiments, the haptic pixels 110are used to modify the haptic data in the haptic track 112, which arethen used to define operation of one or more haptic actuators. In otherembodiments, where the haptic track 112 is encoded in the haptic pixels110, the haptic pixels 110 can define haptic data that are used tocontrol one or more haptic actuators.

As illustrated in FIG. 7, the haptic pixels 110 can be represented as ahaptic texture map 222. The haptic texture map 222 is associated with atleast one of the video texture maps 212. In one possible embodiment, thehaptic texture maps 222 correspond to the video texture maps 212,respectively, such that the temporal resolutions of the haptic texturemaps 222 and the video texture maps 212 are identical. In other possibleembodiments, at least one of the haptic texture maps 222 has a differenttemporal resolution than at least one of the video texture maps 212. Forexample, at least one of the haptic texture maps 222 is associated withtwo or more of the video texture maps 212, such that the haptic texturemaps 222 have a lower temporal resolution than the video texture maps.In other examples, at least one of the video texture maps 212 isassociated with two or more of the haptic texture maps 222, such thatthe haptic texture maps 222 have a higher temporal resolution than thevideo texture maps. An example application of a different temporalresolution is when an image or a representation of something carriesover a plurality of video frames while the image or the representationof something has the same haptic effect over the plurality of videoframes.

In some embodiments, the haptic texture map 222 has the same spatialresolution as the video texture map 212. For example, the number ofhaptic pixels 110 in the haptic texture map 222 is the same as thenumber of video pixels 216 in the video texture map 212. In otherembodiments, the haptic texture map 222 has a different spatialresolution than the video texture map 212. For example, at least one ofthe haptic pixels 110 in the haptic texture map 222 can be associatedwith two or more of the video pixels 216 in the video texture map 212.In the illustrated embodiment, the haptic texture map 222 has half thespatial resolution of the video texture map 212 such that every pair ofvideo pixels 216 can be mapped to a single haptic pixel 110. Asillustrated in FIG. 7, two video pixels 216 identified astwo-dimensional coordinates (x1, y1) and (x2, y2) are associated with asingle haptic pixel 110 identified as two-dimensional coordinate (u1,v1), and the other pairs of video pixels 216 are mapped to other hapticpixels 110 in the same manner. Other configurations are also possible inother embodiments.

In some embodiments, the haptic texture map 222 can be a series ofimages that correspond to key frames. For example, a haptic texture map222 can be arranged between two adjacent video texture maps or twoadjacent video frames. In this configuration, the haptic pixels(including haptic data modifiers) in the haptic texture map can bedetermined based on an interpolation value between the two adjacentvideo texture maps or video frames that bracket a timestamp for thehaptic texture map.

In at least some embodiments, the haptic texture map is a monochromaticimage, each pixel of which has a value representative of a haptic datamodifier associated with that pixel. For example, the haptic texture mapcan be a greymap. In one example, each pixel can have a value of 8-bitdata, ranging from 0 to 255. Other data types can also be possible, suchas 16-bit, 32-bit, etc. The pixels can have different valuesrepresentative of different haptic data modifiers. In other embodiments,the haptic texture map is an image of multiple colors for representingmultiple haptic data modifiers for each pixel.

Referring to FIG. 8, in one possible embodiment, the haptic pixel 110can include a pixel identifier 230 and one or more haptic effectmodifiers 232. The pixel identifier 230 can be used to identify eachhaptic pixel. In at least some embodiments, the pixel identifierincludes a coordinate that identifies the position of the associatedhaptic pixel 110 among the other haptic pixels 110 (or within the haptictexture map 222). The coordinates of the haptic pixels 110 can be usedto map between the video pixels 216 in the video texture map 212 and thehaptic pixels 110 in the corresponding haptic texture map 222. Forexample, as illustrated in FIG. 7, the haptic pixels 110 havetwo-dimensional coordinates (u, v), which can be associated with one ormore two-dimensional coordinates (x, y) of the video pixels 216.

The haptic effect modifiers 232 include information used to modify thehaptic track 112. In at least some embodiments, each haptic pixel 110includes, or is associated with, one or more haptic effect modifiers 232which can edit at least a portion of the haptic data (e.g., one or moreparameters used to generate haptic effects) in the haptic track 112,thereby modifying operation of an haptic actuator. In the illustratedembodiment, each haptic pixel 110 is associated with two haptic effectmodifiers 232 (i.e., a first haptic effect modifier 232A and a secondhaptic effect modifier 232B). A plurality of haptic effect modifiers 232in each haptic pixel 110 are used to modify different parameters in thehaptic data in the haptic track 112.

In one possible embodiment, the haptic effect modifiers 232 includeattenuation data for adjusting magnitude (amplitude) of haptic effects.In other possible embodiments, the haptic effect modifiers 232 includedata for adjusting other parameters including frequency, waveform,sharpness, and other possible attributes of a haptic effect, andselection of one or more actuators for generating a haptic effect.

For non-monochromatic encoding for a haptic texture map, each of thehaptic data modifiers needs to be mapped to an image property. Forexample, where each haptic pixel includes two haptic data modifiers fordefining two parameters, two colors in RGB color values can be used todefine the two haptic data modifiers, such that a first haptic datamodifier has a red pixel value and a second haptic data modifier has agreen pixel value. Other combinations of colors (e.g., red and blue, orgreen and blue) are also possible. Where each haptic pixel includesthree haptic data modifiers for defining three parameters, RGB colorvalues can be used to define the three haptic data modifiers, such thata first haptic data modifier has a red pixel value, a second haptic datamodifier has a green pixel value, and a third haptic data modifier has ablue pixel value. The mapping using color values can be limited by theexisting encoding ranges because, in the above example, RGB color systemcan only support three haptic data modifiers. In other embodiments whereencoding formats can support layers, each of a plurality of haptic datamodifiers can be a monochromatic image in a separate layer. This mayenable an unlimited number of haptic data modifiers for each hapticpixel.

In some possible embodiments, the haptic effect modifiers 232 can berepresented as numerical values. By way of example, each haptic effectmodifier 232 has values of 8 bits, such as values ranging from 0 to 255.In other possible embodiments, the haptic effect modifiers 232 can beexpressed in different ways.

FIG. 9 is a flowchart illustrating an exemplary method 240 forgenerating haptic data modifiers 232. In the illustrated embodiment, atoperation 242, the haptic effect authoring device 102 can display thehaptic texture map 222 on a display device, such as the user interface150 (e.g., the output device 164), of the haptic effect authoring device102. Then, at operation 244, the haptic effect authoring device 102receives one or more haptic data modifier values via an input device,such as the input device 162, of the haptic effect authoring device 102.At operation 246, the haptic effect authoring device 102 can thendisplay a modified haptic texture map 222 which has been updated torepresent the input of the haptic data modifier values.

Referring to FIG. 10, one possible embodiment of a user interface 250 isillustrated for receiving a user input of haptic data modifier values.The user interface 250 can be implemented using the user interface 150of the haptic effect authoring device 102. In this example embodiment,the user interface 250 includes a map display segment 252, an areaselection segment 254, and one or more parameter selection segments 256(including 256A and 256B).

It is understood that the user interface 250 is not limited to one asillustrated in FIG. 10. The user interface 250 can be any user interfaceof existing image and video editing software or tools. In someembodiments, the haptic effect authoring device 102 of the presentdisclosure can be incorporated in, or implemented using, one or moreexisting image and video editing tools, and therefore the user interface250 may be implemented by a user interface of such existing image andvideo editing tools.

The map display segment 252 is used to display the haptic texture map222. In at least some embodiments, the haptic texture map 222 displayedin the map display segment 252 can at least partially change asdifferent haptic data modifier values are entered.

The area selection segment 254 is used to select and/or display an areain the haptic texture map 222, which can be selected by a user. Once thearea is selected on the haptic texture map 222, the user can input oneor more haptic data modifier values to adjust haptic effects associatedwith the haptic pixels within the selected area. The area can be anindividual haptic pixels or a group of haptic pixels. Various methodscan be used to enable the user to select an area in the haptic texturemap 222. For example, the user can use a peripheral input device (e.g.,the mouse) to define the area, such as selecting one or more hapticpixels individually or drawing a region (e.g., rectangular or circular)on the haptic texture map 222 to select a plurality of haptic pixels.

The parameter selection segments 256 can present control elements 258(including 258A and 258B) for enabling the user to adjust haptic datamodifier values. Various types of control elements 258 can be used, suchas buttons, sliders, list boxes, spinners, and drop-down lists. In theillustrated embodiments, the first parameter selection segment 256A andthe second parameter selection segment 256B include a first slider 258Aand a second slider 258B, so that the user can change the position ofhandles along bars to select the haptic data modifier values (e.g.,Parameter 1 in the first parameter selection segment 256A and Parameter2 in the second parameter selection segment 256B). In other embodiments,the user interface 250 has different configurations, arrangements, andlayouts than illustrated herein.

Referring now to FIGS. 11-15, one possible exemplary method forrendering haptic effects for the content played back based on thecontent data 114. This method can implement the operation 134 asdescribed in FIG. 2.

FIG. 11 illustrates a block diagram of one of many possible embodimentsof a haptic enabled apparatus 104 as illustrated in FIG. 1. The hapticenabled apparatus 104 can be of various configurations. The hapticenabled apparatus 104 can be any type of device that can be used todeliver haptic effects, such as a cellular phone, a smart phone, apersonal digital assistant (PDA), a portable music player, a portablevideo player, a game system, a virtual reality (VR) system, a virtualreality headset, a 360-degree video headset, an automotive system, anavigation system, a desktop, a laptop computer, electronic appliances(e.g., a television, an oven, a washer, a dryer, a refrigerator, or alighting system), a movie theater such as IMAX™ theater with seats,headsets, or other devices having haptic actuators, and any otherelectronic or computing devices capable of processing information aswell as providing haptic feedback.

The haptic enabled apparatus 104 includes an input device 302, aprocessor 304, memory 306, an actuator drive circuit 308, and anactuator 310. In some embodiments and as illustrated in FIG. 11, theinput device 302, the processor 304, the memory 306, the actuator drivecircuit 308, and the actuator 310 are incorporated into a single device,which can be worn or carried by a user. In other embodiments, at leastone of the input device 302, the processor 304, the memory 306, theactuator drive circuit 308, and the actuator 310 is separately arrangedfrom the others and connected to each other either wirelessly or bywire.

The input device 302 is configured to monitor or detect one or moreevents associated with the haptic enabled apparatus 104 or a user of thehaptic enabled apparatus 104, or one or more events performed by theuser, of which the user can be informed with a haptic feedback. Theinput device 302 is any device that inputs a signal into the processor304.

An example of an input device 302 is a touch sensitive surface or othertype of user interface mounted within a housing of the device 104, suchas a mouse, touchpad, mini-joystick, scroll wheel, trackball, game padsor game controllers. Another example of an input device 302 is a controldevice such as a key, button, switch or other type of user interfaces.Yet another example of an input device 302 is a transducer that inputs asignal into the processor 304. Examples of transducers that can be usedas an input device 302 include one or more antennas and sensors.

A sensor can be any instrument or other device that outputs a signal inresponse to receiving a stimulus. The sensor can be hardwired to theprocessor or can be connected to the processor wirelessly. The sensorcan be used to detect or sense a variety of different conditions,events, environmental conditions, the operation or condition of thedevice 104, the presence of other people or objects, or any othercondition or thing capable of stimulating a sensor.

Examples of sensors include acoustical or sound sensors such asmicrophones; vibration sensors; chemical and particle sensors such asbreathalyzers, carbon monoxide and carbon dioxide sensors, and Geigercounters; electrical and magnetic sensors such as voltage detectors orhall-effect sensors; flow sensors; navigational sensors or instrumentssuch as GPS receivers, altimeters, gyroscopes, magnetometers oraccelerometers; position, proximity, and movement-related sensors suchas piezoelectric materials, rangefinders, odometers, speedometers, shockdetectors; imaging and other optical sensors such as charge-coupleddevices (CCD), CMOS sensors, infrared sensors, and photodetectors;pressure sensors such as barometers, piezometers, and tactile sensors;force sensors such as piezoelectric sensors and strain gauges;temperature and heat sensors such as thermometers, calorimeters,thermistors, thermocouples, and pyrometers; proximity and presencesensors such as motion detectors, triangulation sensors, radars, photocells, sonars, and hall-effect sensors; biochips; biometric sensors suchas blood pressure sensors, pulse/ox sensors, blood glucose sensors, andheart monitors. Additionally, sensors can be formed with smartmaterials, such as piezo-electric polymers, which in some embodimentsfunction as both a sensor and an actuator.

Various embodiments can include a single input device or can include twoor more input devices. Additionally, various embodiments can includedifferent types of input devices. For example, at least some possibleembodiments include a switch and a transducer such as an antenna or asensor. When the input device 302 is stimulated and inputs a signal tothe processor 304, the processor 304 operates an actuator 310 to providea haptic effect to the person carrying, wearing or interacting with thedevice 104.

The processor 304 can be any device, element, or circuit configured toprocess information and can include any suitable analog or digitalcircuit. The processor 118 also can include a programmable circuit thatexecutes instructions. Examples of programmable circuits includemicroprocessors, microcontrollers, application specific integratedcircuits (ASIC), programmable gate arrays (PLA), field programmable gatearrays (FPGA), or any other processor or hardware suitable for executinginstructions. In various embodiments, the processor 118 can be a singleunit or a combination of two or more units. If the processor 118includes two or more units, the units can be physically located in asingle controller or in separate devices. The processor 304 may be thesame processor that operates the entire device 104, or may be a separateprocessor. The processor 304 can decide what haptic effects are to beplayed and the order in which the effects are played based on high levelparameters. In general, the high level parameters that define aparticular haptic effect include magnitude, frequency and duration. Lowlevel parameters such as streaming motor commands could also be used todetermine a particular haptic effect.

The processor 304 receives signals or data from the input device 302 andoutputs control signals to drive the actuator drive circuit 308. Datareceived by the processor 304 can be any type of parameters,instructions, flags, or other information that is processed by theprocessors, program modules, and other hardware disclosed herein.

The memory device 306 can be any type of storage device orcomputer-readable medium, such as random access memory (RAM), read-onlymemory (ROM), electrically erasable programmable read only memory(EEPROM), flash memory, magnetic memory, optical memory, or any othersuitable memory technology. The memory 306 also can include acombination of volatile and nonvolatile memory. The memory 306 storesinstructions executed by the processor 304. The memory 306 may also belocated internal to the processor 304, or any combination of internaland external memory.

Among the instructions, the memory 306 includes an actuator drive module314 which are instructions that, when executed by the processor 304,generate control signals for the actuator drive circuit 308. Theactuator drive module 314 can also determine feedback from the actuator310 and adjust the control signals accordingly.

The actuator drive circuit 308 is a circuit that receives a hapticsignal (which is also referred to herein as a control signal) from theactuator drive module 314. The haptic signal embodies haptic dataassociated with haptic effects, and the haptic data defines parametersthe actuator drive circuit 308 uses to generate an actuator drivesignal. In exemplary embodiments, such parameters relate to, or areassociated with, electrical characteristics. Examples of electricalcharacteristics that can be defined by the haptic data includesfrequency, amplitude, phase, inversion, duration, waveform, attack time,rise time, fade time, and lag or lead time relative to an event. Theactuator drive signal is applied to the actuator 310 to cause one ormore haptic effects.

The actuator 310, which is also referred to herein as a haptic outputdevice, operates to generate haptic effects. The actuator 310 iscontrolled by the processor 304 that executes the actuator drive module314, which sends a haptic signal to the actuator drive circuit 308. Theactuator drive circuit 308 then generates and applies an actuator drivesignal to the actuator 310 to drive the actuator 310. When applied tothe actuator 310, an actuator drive signal causes the actuator 310 togenerate haptic effects by activating and braking the actuator 310.

The actuator 310 can be of various types. In the illustratedembodiments, the actuator is a resonant actuator, such as a LinearResonant Actuator (LRA) in which a mass attached to a spring is drivenback and forth. In other embodiments, the actuator is a solenoidresonant actuator (SRA).

Other types of electromagnetic actuators are also used, such as anEccentric Rotating Mass (ERM) in which an eccentric mass is moved by amotor or a “smart material” such as piezoelectric, electro-activepolymers or shape memory alloys. Actuators 310 also broadly includenon-mechanical or non-vibratory devices such as those that useelectrostatic friction (ESF), ultrasonic surface friction (USF), orthose that induce acoustic radiation pressure with an ultrasonic haptictransducer, or those that use a haptic substrate and a flexible ordeformable surface, or those that provide projected haptic output suchas a puff of air using an air jet, and so on.

The apparatus 104 may include more than one actuator 310, and eachactuator may include a separate actuator drive circuit 308, all coupledto the processor 304. In embodiments with more than one actuator, eachactuator can have a different output capability in order to create awide range of haptic effects on the device.

As also illustrated in FIG. 1, the haptic enabled apparatus 104 includesthe content presentation device 108 configured to present a contentbased on the content data 114. In the illustrated example, the contentpresentation device 106 includes a display device 316 for presenting avideo content. The display device 316 can be of various configurations,such as display screens or wearable display devices (e.g., head-mounteddisplays).

In the illustrated embodiment, the haptic effect rendering device 106 asillustrated in FIG. 1 may include at least one of the devices andelements in the haptic enabled apparatus 104. For example, the hapticeffect rendering device 106 can include the processor 304, the memory306 including the actuator drive module 314, and the actuator drivecircuit 308. In other examples, the haptic effect rendering device 106can include more or less devices and elements than illustrated in FIG.11.

FIGS. 12-16 illustrate an exemplary method 350 for generating hapticeffects. In FIG. 12, in some embodiments, the method 350 is performed bythe haptic enabled apparatus 104. In other embodiments, the method 350can be performed using other computing devices either associated with orindependent from the haptic enabled apparatus 104.

At operation 352, the haptic enabled apparatus 104 receives a haptictrack 112 and a plurality of haptic pixels 110. As described herein, thehaptic track 112 includes haptic data defining operation of a hapticactuator 310. The haptic pixels 110 include haptic data modifiers 232.The haptic pixels 110 are associated with the video pixels 216 in thevideo texture map 212 that represents at least one frame 218 of a video214.

At operation 354, the apparatus 104 operates to synchronize the haptictrack 112 with the video 214 as the video is played back. In at leastsome possible embodiments, the apparatus 104 obtains timestamps of thevideo 214 and the haptic track 112 and synchronize the haptic track 112with the frames of the video based on the timestamps. Other methods forsynchronizing the haptic track 112 with the video 214 are also possiblein other embodiments.

At operation 356, the apparatus 104 operates to identify a point ofinterest 320 in the frame 218 of the video 214. The point of interest320 can be one or more of various points in the video frame 218. In someembodiments, the point of interest 320 is a point at which a user orviewer looks in the video frame 218. In other embodiments, the point ofinterest 320 is a point at which a user or viewer positions a pointer orcursor on a display device presenting the video frame 218. Where thevideo 214 is a 360-degree video or a three-dimensional video, the pointof interest 320 can be represented as a three-dimensional vector, suchas a user perspective vector 322 (also referred to herein as a lookvector) as illustrated in FIG. 13.

The point of interest 320 can include one of the video pixels 216 in thevideo texture map 212 corresponding to the video frame 218. The videopixel 216 that represents the point of interest 320 can be identified asa target video pixel 324. In possible other embodiments, the point ofinterest 320 includes a plurality of video pixels 216 in the videotexture map 212. An example method of identifying the point of interest320 is further described with reference to FIG. 14.

At operation 358, the apparatus 104 operates to determine a targethaptic pixel 330 (FIG. 15) based on the point of interest 320. Thetarget haptic pixel 330 is identified as one of the haptic pixels 110 inthe haptic texture map 222 that corresponds to the target texture map212. In at least some embodiment, the target haptic pixel 330corresponds to the target video pixel 324 in the corresponding targettexture map 212. For example, the target haptic pixel 330 is determinedby mapping the target video pixel 324 in the video texture map 212 to acorresponding haptic pixel in the haptic texture map 222. In otherpossible embodiments, the target haptic pixels 330 is identified as twoor more of the haptic pixels 110 that correspond to the target videopixel or pixels 324.

At operation 360, the apparatus 104 obtains a target haptic datamodifier 334 associated with the target haptic pixel 330. As illustratedin FIG. 16, the target haptic data modifier 334 can be determined byidentifying a coordinate 230 of the target haptic pixel 330 andretrieving the haptic data modifier 232 in the coordinate 230. Where thehaptic pixels 110 have a plurality of haptic data modifiers 232, aplurality of target haptic data modifiers 334 are obtained accordingly.

At operation 362, the apparatus 104 operates to modify the haptic datain the haptic track 112 based on the target haptic data modifier 334. Asillustrated in FIG. 16, in some embodiments, a modified haptic track 340can be generated having the modified haptic data.

One or more parameters in the haptic data can be adjusted based on whatparameter(s) the target haptic data modifier 334 relates to. Asdescribed herein, the target haptic data modifier 334 can modify, forexample, magnitude (amplitude), frequency, waveform, sharpness, andother possible attributes of a haptic effect.

At operation 364, the apparatus 104 operates to control operation of thehaptic actuator 310 based on the modified haptic track 340 including themodified haptic data. In other possible embodiments, a plurality ofhaptic actuators 310 can be either simultaneously or individuallycontrolled based on one or more modified haptic tracks 340.

Referring to FIG. 14, an example method 370 of identifying the point ofinterest 320 is described. In some embodiments, the method 370 can beperformed by the haptic enabled apparatus 104. In other embodiments, themethod 370 can be performed using other computing devices eitherassociated with or independent from the haptic enabled apparatus 104.

At operation 372, the haptic enabled apparatus 104 determines a userperspective vector 322. In at least some embodiments, the userperspective vector 322 is a three-dimensional vector, which is modeledas a camera or eye at the origin 380 of a three-dimensional space orpolyhedron 382. In the illustrated embodiment, the user perspectivevector 322 can be represented with a vector V(d_(x), d_(y), d_(z)).

In possible embodiments, the video texture map 212 can be projected tothe three-dimensional space 382 surrounding the origin 380. In someembodiments, the video texture map 212 is projected to a sphere asillustrated in FIG. 13. In other embodiments, the video texture map 212can be projected to a non-spherical polyhedron.

At operation 374, the apparatus 104 determines an intersection 384between the user perspective vector 322 and the projection of the videotexture map 212 on the three-dimensional space 382.

At operation 376, the apparatus 104 identifies the intersection 384 asthe target video pixel 324 in the video texture map 212. By way ofexample, where the three-dimensional space 382 is a sphere, thecoordinate (u, v) of the target video pixel 324 (or the intersection384) can be calculated by:

$u = {{0.5 + {\frac{\arctan \; 2\left( {d_{z},d_{x}} \right)}{2\; \pi}\mspace{14mu} v}} = {0.5 - {\frac{\arcsin \; \left( d_{y} \right)}{\pi}.}}}$

At operation 378, the apparatus 104 operates to determine the targethaptic pixel 330 based on the target video pixel 324. As describedabove, the target haptic pixel 330 is determined by mapping the targetvideo pixel 324 in the video texture map 212 to a corresponding hapticpixel in the haptic texture map 222. As shown in FIG. 16, the targethaptic pixel 330 provides one or more target haptic data modifiers 334.

Referring to FIG. 15, an exemplary process of rendering a video contentwith haptic effects is described. As illustrated in FIG. 6, a video,such as a 360-degree video, can be played back by displaying a series ofvideo frames 218 at a determined frame rate. The video frames 218 can bepresented by a plurality of video texture maps 212. Typically, eachvideo frame 218 is presented by a single video texture map 212. However,in some embodiments, a single video texture map 212 can implement two ormore video frames 218. Therefore, the temporal resolution of the videoframes 218 can be identical to, or different from, the temporalresolution of the video texture maps 212. As the video frames 218 aredisplayed, one or more haptic tracks 112 are executed to provide hapticeffects. In the illustrated embodiments, two haptic tracks 112A and 112Bare run. As described herein, one or more haptic texture maps 222 areused to modify the haptic data in the haptic tracks. For example, thehaptic texture maps 222 include one or more haptic data modifiers foreach haptic pixel. The haptic data modifiers are used to modify thehaptic data in the haptic tracks. Since a plurality of haptic pixels ineach haptic texture map may have different sets of haptic datamodifiers, the haptic effects can be localized to a specific object atwhich a point of interest, such as a viewer's perspective, is located.As described herein, the spatial and/or temporal resolutions can be thesame or different between the haptic texture maps and the video texturemaps.

Referring now to FIG. 17-18, another exemplary method for generatinghaptic effects is described.

FIG. 17 illustrates a data structure of another exemplary set of hapticpixels 110. In this embodiment, the haptic pixels 110 include hapticdata for controlling one or more haptic actuators 310. In thisconfiguration, haptic effects can be rendered using the haptic pixels110 without requiring a haptic track 112. In effect, the haptic track112 is encoded in the haptic pixels 110.

In the illustrated embodiment, each haptic pixel 110 includes a pixelidentifier 230 and one or more haptic parameters 402. Similarly to thehaptic pixels 110 in FIG. 8, the pixel identifier 230 is used toidentify each haptic pixel. In at least some embodiments, the pixelidentifier includes a coordinate that identifies the position of theassociated haptic pixel 110 among the other haptic pixels 110 (or withinthe haptic texture map 222).

The haptic parameters 402 are used to generate signals for driving oneor more haptic actuators to create haptic effects. Examples of theparameters include amplitude, frequency, waveform, sharpness, and otherpossible attributes of a haptic effect, and selection of one or moreactuators for generating a haptic effect.

FIG. 18 is a possible exemplary method 450 for generating haptic effectsusing the haptic pixels 110 of FIG. 17. In some embodiments, the method450 is performed by the haptic enabled apparatus 104. In otherembodiments, the method 450 can be performed using other computingdevices either associated with or independent from the haptic enabledapparatus 104. The method 450 is similar to the method 300 of FIG. 12except for the haptic pixels 110. Therefore, the details of theoperations in the method 450 are not repeated for brevity to the extentpossible.

At operation 452, the haptic enabled apparatus 104 receives a pluralityof haptic pixels 110 as described in FIG. 17. Each haptic pixel isassociated with at least one video pixel 216 and includes haptic data.The haptic data defines operation of one or more haptic actuators. Thevideo pixel 216 is included in a video texture map 212 representative ofat least one frame 218 of a video 214.

At operation 454, the haptic enabled apparatus 104 identifies a point ofinterest 320 in the frame 218 of the video. The point of interest 320includes a target video pixel 234 in the frame of the video.

At operation 456, the haptic enabled apparatus 104 determines a targethaptic pixel 330 among the haptic pixels 110. The target haptic pixel330 corresponds to the target video pixel 234 in the frame of the video.

At operation 458, the haptic enabled apparatus 104 obtains target hapticdata 470 included in the target haptic pixel 330. The target haptic data470 is data included in the target haptic pixel 330 and include hapticparameters 402 as described in FIG. 17. In this embodiment, the hapticsignal can be synthesized directly from data in the haptic pixel,instead of modulating a pre-existing signal such as a separate haptictrack.

At operation 460, the haptic enabled apparatus 104 operates to controloperation of one or more haptic actuator based on the target haptic data470.

As such, the present disclosure provides a simple method for authoringand encoding haptic effects by exploiting existing video toolchains andencodings, and enables a video-to-haptic conversion at least forspatialization. According to the present disclosure, haptic effects canbe localized to a specific object in a video scene or frame by filteringa source video for a specific object using a video editing tool. Suchlocalization can then be modulated in time, using a pre-existing haptictime series designed for that object.

The various examples and teachings described above are provided by wayof illustration only and should not be construed to limit the scope ofthe present disclosure. Those skilled in the art will readily recognizevarious modifications and changes that may be made without following theexamples and applications illustrated and described herein, and withoutdeparting from the true spirit and scope of the present disclosure.

What is claimed is:
 1. A method of authoring haptic effects, the methodcomprising: receiving a haptic track and a video texture map, the haptictrack having haptic data, the haptic data defining operation of a hapticactuator, the video texture map representative of at least one frame ofa video, the haptic track associated with the at least one frame of thevideo, and the video texture map having a plurality of video pixels; andgenerating haptic data modifiers associated with haptic pixels, eachhaptic pixel associated with at least one of the video pixels, thehaptic data modifiers for modifying at least a portion of the hapticdata and modifying operation of the haptic actuator.